{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1fe6e643-9453-4381-9445-bd471685fb96",
   "metadata": {},
   "source": [
    "## Exploring the company match dataset using Autolabel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80110a5b-2b3e-45e2-a2da-f6fa00200dff",
   "metadata": {},
   "source": [
    "#### Setup the API Keys for providers that you want to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "92993c83-4473-4e05-9510-f543b070c7d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# provide your own OpenAI API key here\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "01b8fcdd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading example dataset from https://autolabel-benchmarking.s3.us-west-2.amazonaws.com/painting-style-classification/seed.csv to seed.csv...\n",
      "Downloading example dataset from https://autolabel-benchmarking.s3.us-west-2.amazonaws.com/painting-style-classification/test.csv to test.csv...\n",
      "100% [........................................] [57543/57543] bytes\r"
     ]
    }
   ],
   "source": [
    "from autolabel import get_data\n",
    "\n",
    "get_data(\"painting-style-classification\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84b014d1-f45c-4479-9acc-0d20870b1786",
   "metadata": {},
   "source": [
    "## Start the labeling process!\n",
    "\n",
    "Labeling with Autolabel is a 3-step process:\n",
    "* First, we specify a labeling configuration (see `config.json` below)\n",
    "* Next, we do a dry-run on our dataset using the LLM specified in `config.json` by running `agent.plan`\n",
    "* Finally, we run the labeling with `agent.run`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ababbe3d",
   "metadata": {},
   "source": [
    "### First labeling run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c093fe91-3508-4140-8bd6-217034e3cce6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "from autolabel import LabelingAgent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c93fae0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the config\n",
    "with open(\"image_classification.json\") as f:\n",
    "     config = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2448ffd6",
   "metadata": {},
   "source": [
    "Let's review the configuration file below. You'll notice the following useful keys:\n",
    "* `task_type`: `entity_matching` (since it's an entity matching task)\n",
    "* `model`: `{'provider': 'openai', 'name': 'gpt-3.5-turbo'}` (use a specific OpenAI model)\n",
    "* `prompt.task_guidelines`: `'You are provided with descriptions of companies from their websites...` (how we describe the task to the LLM)\n",
    "* `prompt.labels`: `['not duplicate', 'duplicate']` (the full list of labels to choose from)\n",
    "* `prompt.few_shot_num`: 3 (how many labeled examples to provide to the LLM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "31c4b617",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'task_name': 'ImageClassification',\n",
       " 'task_type': 'classification',\n",
       " 'dataset': {'label_column': 'label',\n",
       "  'delimiter': ',',\n",
       "  'image_url_column': 'image_url'},\n",
       " 'model': {'provider': 'openai_vision', 'name': 'gpt-4-vision-preview'},\n",
       " 'prompt': {'task_guidelines': \"Given the description of a painting, predict the style of the paining. You will be first shown multiple descriptions and their styles. For the last input, you'll be shown an image along with the description and your job is to predict the style for this input. Your answer must be from one of the following categories:\\n{labels}\",\n",
       "  'labels': ['Impressionism',\n",
       "   'Color Field Painting',\n",
       "   'Early Renaissance',\n",
       "   'Fauvism',\n",
       "   'Minimalism',\n",
       "   'Romanticism',\n",
       "   'Mannerism Late Renaissance',\n",
       "   'Post Impressionism',\n",
       "   'Contemporary Realism',\n",
       "   'Pointillism',\n",
       "   'Ukiyo e',\n",
       "   'Abstract Expressionism',\n",
       "   'Analytical Cubism',\n",
       "   'Art Nouveau Modern',\n",
       "   'Expressionism',\n",
       "   'High Renaissance',\n",
       "   'Cubism',\n",
       "   'Naive Art Primitivism',\n",
       "   'Rococo',\n",
       "   'Pop Art',\n",
       "   'Synthetic Cubism',\n",
       "   'Realism',\n",
       "   'Symbolism',\n",
       "   'Northern Renaissance',\n",
       "   'Baroque',\n",
       "   'New Realism'],\n",
       "  'few_shot_examples': 'seed.csv',\n",
       "  'few_shot_selection': 'semantic_similarity',\n",
       "  'few_shot_num': 5,\n",
       "  'example_template': 'Input: Description: {description} \\nOutput: {label}'}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "acb4a3de-fa84-4b94-b17a-7a6fac892a1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an agent for labeling\n",
    "agent = LabelingAgent(config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "92667a39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "013bca8159494c3187fdf817628d828e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┌──────────────────────────┬─────────┐\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Total Estimated Cost     </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $5.3897 </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Number of Examples       </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> 259     </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Average cost per example </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $0.0208 </span>│\n",
       "└──────────────────────────┴─────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┌──────────────────────────┬─────────┐\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mTotal Estimated Cost    \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$5.3897\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mNumber of Examples      \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m259    \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mAverage cost per example\u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$0.0208\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "└──────────────────────────┴─────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────── </span>Prompt Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────── \u001b[0mPrompt Example\u001b[92m ──────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">\"text\"</span>: <span style=\"color: #008000; text-decoration-color: #008000\">\"Given the description of a painting, predict the style of the paining. You will be first shown multiple </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">descriptions and their styles. For the last input, you'll be shown an image along with the description and your job</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">is to predict the style for this input. Your answer must be from one of the following </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">categories:\\nImpressionism\\nColor Field Painting\\nEarly Renaissance\\nFauvism\\nMinimalism\\nRomanticism\\nMannerism </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Late Renaissance\\nPost Impressionism\\nContemporary Realism\\nPointillism\\nUkiyo e\\nAbstract </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Expressionism\\nAnalytical Cubism\\nArt Nouveau Modern\\nExpressionism\\nHigh Renaissance\\nCubism\\nNaive Art </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Primitivism\\nRococo\\nPop Art\\nSynthetic Cubism\\nRealism\\nSymbolism\\nNorthern Renaissance\\nBaroque\\nNew </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Realism\\n\\nYou will return the answer with just one element: \\\"the correct label\\\"\\n\\nSome examples with their </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">output answers are provided below:\\n\\nInput: Description: maurice utrillo la butte pinson 1 \\nOutput: Post </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Impressionism\\nInput: Description: vincent van gogh le moulin de la galette 1886 1 \\nOutput: Post </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Impressionism\\nInput: Description: maurice utrillo vase with flowers 1 \\nOutput: Post Impressionism\\nInput: </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">Description: maurice de vlaminck suresnes \\nOutput: Cubism\\nInput: Description: maurice esteve broucl 1973 1 </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">\\nOutput: Color Field Painting\\n\\nNow I want you to label the following example:\\nInput: Description: maurice </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">utrillo moulin de la galette 3 \\nOutput: \"</span>, <span style=\"color: #008000; text-decoration-color: #008000\">\"image_url\"</span>: \n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">\"https://autolabel-benchmarking.s3.amazonaws.com/painting-style-classification/maurice-utrillo_moulin-de-la-galette</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">-3_jpg.rf.7572fbf1699f459fc883c19e01caf82a.jpg\"</span><span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\u001b[32m\"text\"\u001b[0m: \u001b[32m\"Given the description of a painting, predict the style of the paining. You will be first shown multiple \u001b[0m\n",
       "\u001b[32mdescriptions and their styles. For the last input, you'll be shown an image along with the description and your job\u001b[0m\n",
       "\u001b[32mis to predict the style for this input. Your answer must be from one of the following \u001b[0m\n",
       "\u001b[32mcategories:\\nImpressionism\\nColor Field Painting\\nEarly Renaissance\\nFauvism\\nMinimalism\\nRomanticism\\nMannerism \u001b[0m\n",
       "\u001b[32mLate Renaissance\\nPost Impressionism\\nContemporary Realism\\nPointillism\\nUkiyo e\\nAbstract \u001b[0m\n",
       "\u001b[32mExpressionism\\nAnalytical Cubism\\nArt Nouveau Modern\\nExpressionism\\nHigh Renaissance\\nCubism\\nNaive Art \u001b[0m\n",
       "\u001b[32mPrimitivism\\nRococo\\nPop Art\\nSynthetic Cubism\\nRealism\\nSymbolism\\nNorthern Renaissance\\nBaroque\\nNew \u001b[0m\n",
       "\u001b[32mRealism\\n\\nYou will return the answer with just one element: \\\"the correct label\\\"\\n\\nSome examples with their \u001b[0m\n",
       "\u001b[32moutput answers are provided below:\\n\\nInput: Description: maurice utrillo la butte pinson 1 \\nOutput: Post \u001b[0m\n",
       "\u001b[32mImpressionism\\nInput: Description: vincent van gogh le moulin de la galette 1886 1 \\nOutput: Post \u001b[0m\n",
       "\u001b[32mImpressionism\\nInput: Description: maurice utrillo vase with flowers 1 \\nOutput: Post Impressionism\\nInput: \u001b[0m\n",
       "\u001b[32mDescription: maurice de vlaminck suresnes \\nOutput: Cubism\\nInput: Description: maurice esteve broucl 1973 1 \u001b[0m\n",
       "\u001b[32m\\nOutput: Color Field Painting\\n\\nNow I want you to label the following example:\\nInput: Description: maurice \u001b[0m\n",
       "\u001b[32mutrillo moulin de la galette 3 \\nOutput: \"\u001b[0m, \u001b[32m\"image_url\"\u001b[0m: \n",
       "\u001b[32m\"https://autolabel-benchmarking.s3.amazonaws.com/painting-style-classification/maurice-utrillo_moulin-de-la-galette\u001b[0m\n",
       "\u001b[32m-3_jpg.rf.7572fbf1699f459fc883c19e01caf82a.jpg\"\u001b[0m\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# dry-run -- this tells us how much this will cost and shows an example prompt\n",
    "from autolabel import AutolabelDataset\n",
    "\n",
    "ds = AutolabelDataset(\"data/painting-style-classification/test.csv\", config=config)\n",
    "agent.plan(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "dd703025-54d8-4349-b0d6-736d2380e966",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b80c345417a43b4b73ebbf059a37c17",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/dhruva/refuel-ai/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/Users/dhruva/refuel-ai/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "/Users/dhruva/refuel-ai/lib/python3.9/site-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">classification_report:\n",
       "                    precision    recall  f1-score   support\n",
       "\n",
       "Art Nouveau Modern       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.00</span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>\n",
       "     Expressionism       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.33</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.50</span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>\n",
       "     Impressionism       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.67</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.80</span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>\n",
       "Post Impressionism       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.50</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.67</span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>\n",
       "           Realism       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.50</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.67</span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>\n",
       "            Rococo       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>\n",
       "       Romanticism       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>\n",
       "\n",
       "          accuracy                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.70</span>        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>\n",
       "         macro avg       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.71</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.71</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.66</span>        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>\n",
       "      weighted avg       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.77</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.70</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.68</span>        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "classification_report:\n",
       "                    precision    recall  f1-score   support\n",
       "\n",
       "Art Nouveau Modern       \u001b[1;36m0.00\u001b[0m      \u001b[1;36m0.00\u001b[0m      \u001b[1;36m0.00\u001b[0m         \u001b[1;36m1\u001b[0m\n",
       "     Expressionism       \u001b[1;36m0.33\u001b[0m      \u001b[1;36m1.00\u001b[0m      \u001b[1;36m0.50\u001b[0m         \u001b[1;36m1\u001b[0m\n",
       "     Impressionism       \u001b[1;36m0.67\u001b[0m      \u001b[1;36m1.00\u001b[0m      \u001b[1;36m0.80\u001b[0m         \u001b[1;36m2\u001b[0m\n",
       "Post Impressionism       \u001b[1;36m1.00\u001b[0m      \u001b[1;36m0.50\u001b[0m      \u001b[1;36m0.67\u001b[0m         \u001b[1;36m2\u001b[0m\n",
       "           Realism       \u001b[1;36m1.00\u001b[0m      \u001b[1;36m0.50\u001b[0m      \u001b[1;36m0.67\u001b[0m         \u001b[1;36m2\u001b[0m\n",
       "            Rococo       \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m         \u001b[1;36m1\u001b[0m\n",
       "       Romanticism       \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m         \u001b[1;36m1\u001b[0m\n",
       "\n",
       "          accuracy                           \u001b[1;36m0.70\u001b[0m        \u001b[1;36m10\u001b[0m\n",
       "         macro avg       \u001b[1;36m0.71\u001b[0m      \u001b[1;36m0.71\u001b[0m      \u001b[1;36m0.66\u001b[0m        \u001b[1;36m10\u001b[0m\n",
       "      weighted avg       \u001b[1;36m0.77\u001b[0m      \u001b[1;36m0.70\u001b[0m      \u001b[1;36m0.68\u001b[0m        \u001b[1;36m10\u001b[0m\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Actual Cost: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Actual Cost: \u001b[1;36m0.0\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.7      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 10      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1.0             </span>│\n",
       "└──────────┴─────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.7     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m10     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1.0            \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└──────────┴─────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# now, do the actual labeling\n",
    "ds = agent.run(ds, max_items=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52281bb0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "vscode": {
   "interpreter": {
    "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
